Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: bulk streamed processing #22

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

Conversation

stevenh
Copy link
Contributor

@stevenh stevenh commented Jan 22, 2022

Add streamed processing of bulk, eliminating the limits needed to prevent memory requirements as well as recreation of svrquery.Client for each processed element by implementing a reusable BulkClient.

This changes the processing to best effort and can result in entries without an address being output.

Also:

  • Output to a provided stream.
  • Eliminate the use of log.Fatal in the library.
  • Include element details when parsing fails.

Stephen Birch and others added 10 commits November 29, 2021 16:53
…dd-encryption-to-tf2e

� Conflicts:
�	cmd/cli/main.go
�	go.sum
Add streamed processing of bulk, eliminating the limits needed to
prevent memory requirements as well as recreation of svrquery.Client for
each processed element by implementing a reusable BulkClient.

This changes the processing to best effort and can result in entries
without an address being output.

Also:
* Output to a provided stream.
* Eliminate the use of log.Fatal in the library.
* Include element details when parsing fails.
@stevenh stevenh force-pushed the feat/bulk-stream-processing branch from de8d660 to 33ed3b7 Compare January 22, 2022 00:59
Base automatically changed from feat/add-encryption-to-tf2e to master January 24, 2022 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants